Devices and methods for high dynamic range video

ABSTRACT

Systems and methods of the invention merge information from multiple image sensors to provide a high dynamic range (HDR) video. The present invention provides for real-time HDR video production using multiple sensors and pipeline processing techniques. According to the invention, multiple sensors with different exposures each produces an ordered stream of frame-independent pixel values. The pixel values are streamed through a pipeline on a processing device. The pipeline includes a kernel operation that identifies saturated ones of the pixel values. The streams of pixel values are merged to produce an HDR video.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/169,006, filed May 31, 2016, which application claims priority andthe benefit of U.S. Provisional Application No. 62/294,820, filed Feb.12, 2016, the contents of which are incorporated by reference.

TECHNICAL FIELD

This disclosure relates to real-time capture and production of highdynamic range video.

BACKGROUND

The human visual system is capable of identifying and processing visualfeatures with high dynamic range. For example, real-world scenes withcontrast ratios of 1,000,000:1 or greater can be accurately processed bythe human visual cortex. However, most image acquisition devices areonly capable of reproducing or capturing low dynamic range, resulting ina loss of image accuracy. The problem is ever more significant in videoimaging.

There are examples of creating High Dynamic Range images by postprocessing images from multiple sensors, each subject to differentexposures. The resulting “blended” image is intended to capture abroader dynamic range than would be possible from a single sensorwithout a post-processing operation.

Conventional approaches to capturing High Dynamic Range images includeusing sensors to acquire image frames of varying exposures and thenprocessing those frames after acquisition (post-processing). . Suchapproaches present significant challenges when filming video. If thesubject, or camera, is moving then no pair of images will includeexactly the same scene. In such cases, after the initial image capture,a post-processing step is required before a High Dynamic Range video canbe produced.

SUMMARY

Methods and devices of the invention process streams of pixels frommultiple sensors in a frame-independent manner, resulting in real-timeHigh Dynamic Range (HDR) video production. The real-time aspect of theinvention is accomplished by analyzing streams of pixels from thevarious sensors without reference to the frame to which those pixelsbelong. Thus, the frame-independent nature of the invention means thatthere is no need to wait for an entire frame of data to be read from asensor before processing pixel data. The result is an HDR video that hasa dynamic range greater than the range that can be obtained using asingle image sensor, typically 8 bits in depth.

The multiple image sensors used in the invention are exposed toidentical scenes with different light levels and each produces anordered stream of pixel values that are processed in real-time andindependent of frame in which they will reside. The video processingpipeline used to produce HDR images includes kernel and merge operationsthat include identifying saturated pixel values and merging streams ofpixel values. The merging operation includes replacing saturated pixelvalues with corresponding pixel values originating from a differentsensor. The resulting product is high dynamic range video output.

Pixel values from multiple image sensors stream through the inventivepipeline and emerge as an HDR video signal without any need to aggregateor store any full images or frames of data as part of the pipeline.Because the kernel and merge operations process pixel values as theystream through the pipeline, no post-processing is required to produceHDR video. As a result, pixel values stream continually through thepipeline without waiting for a complete set of pixels from a sensorbefore being processed. For example, a CMOS sensor has millions ofpixels, but the first pixel values to stream off of that sensor may beprocessed by the kernel operation or even enter the merge operationbefore a pixel value from , e.g., the millionth pixel from that samesensor even reaches the processing device. Thus, the frame-independentpixel processing systems and methods of the invention produce truereal-time HDR video.

Additionally, a synchronization module in the pipeline synchronizes thestreams of pixel values arriving from the sensors. This means that when,for example, the 60^(th) pixel from a first sensor enters the kerneloperation, the 60^(th) pixel from each of the other sensors is alsosimultaneously entering the kernel operation. As a result, pixel valuesfrom corresponding pixels on different sensors flow through the pipelinesynchronously. This allows two things. First, the synchronization modulecan correct small phase discrepancies in data arrival times to thesystem from multiple sensors.). Second, the synchronization allows thekernel operation to consider—for a given pixel value from a specificpixel on one of the image sensors—values from the neighborhood ofsurrounding pixels on that sensor and also consider values from acorresponding neighborhood of pixels on another of the image sensors.This allows the kernel operation to create an estimated value for asaturated pixel from one sensor based on a pattern of values from thesurrounding neighborhood on the same or another sensor.

The HDR pipeline can also correct for differences in spectralcharacteristics of each of the multiple sensors. Optical components suchas beamsplitters, lenses, or filters—even if purported to be spectrallyneutral—may have slight wavelength-dependent differences in the amountsof light transmitted. That is, each image sensor may be said to have itsown “color correction space” whereby images from that sensor need to becorrected out of that color correction space to true color. The opticalsystem can be calibrated (e.g., by taking a picture of a calibrationcard) and a color correction matrix can be determined and stored foreach image sensor. The HDR video pipeline can then perform thecounter-intuitive step of adjusting the pixel values from one sensortoward the color correction space of another sensor—which may in somecases involve nudging the colors away from true color. This may beaccomplished by multiplying a vector of RGB values from the one sensorby the inverse of the color correction matrix of the other sensor. Afterthis color correction to the second sensor, the streams are merged, andthe resulting HDR video signal is color corrected to true color (e.g.,by multiplying the RGB vectors by the applicable color correctionmatrix). This operation accounts for spectral differences of each imagesensor.

A preferred pipeline includes other processing modules as described indetail in the Detailed Description of the invention below.

In certain aspects, the invention provides methods for producingreal-time HDR video. Methods include streaming pixel values from each ofmultiple sensors in a frame independent-manner through a pipeline on aprocessing device, such as a field-programmable gate array or anapplication-specific integrated circuit. The pipeline includes a kerneloperation that identifies saturated pixel values. The pixel values aremerged to produce an HDR image. The merging step excludes at least someof the saturated pixel values from the resulting HDR image. The multipleimage sensors preferably capture images simultaneously through a singlelens. Incoming light is received through the lens and split via at leastone beamsplitter onto the multiple image sensors, such that at leastabout 95% of the light gathered by the lens is captured by the multipleimage sensors. The multiple image sensors may include at least a highexposure (HE) sensor and a middle exposure (ME) sensor. Merging pixelstreams may include using HE pixel values that are not saturated and MEpixel values corresponding to the saturated pixel values. The multiplesensors may further include a low exposure (LE) sensor, and the methodmay include identifying saturated pixel values originating from both theHE sensor and the ME sensor. The obtaining, streaming, and merging stepsmay include streaming the sequences of pixel values through a pipelineon the processing device such that no location on the processing devicestores a complete image.

Sequences of pixel values are streamed through the processing device andmerged without waiting to receive pixel values from all pixels on theimage sensors. Because the pixel values are streamed through a pipeline,at least some of the saturated pixel values may be identified beforereceiving values from all pixels from the image sensors. Methods of theinvention may include beginning to merge portions of the sequences whilestill streaming later-arriving pixel values through the kerneloperation.

In some embodiments, streaming the pixel values through the kerneloperation includes examining values from a neighborhood of pixelssurrounding a first pixel on the HE sensor, finding saturated values inthe neighborhood of pixels, and using information from a correspondingneighborhood on the ME sensor to estimate a value for the first pixel.The pixel values may be streamed through the kernel operation via a pathwithin the processing device that momentarily stores a value from thefirst pixel proximal to each value originating from the neighborhood ofpixels.

Each of the image sensors may include a color filter array, and themethod may include demosaicing the HDR imaging after the merging. Themultiple image sensors preferably capture images that are opticallyidentical except for light level.

A first pixel value from a first pixel on one of the image sensors maybe identified as saturated if it is above some specified level, forexample at least about 90% of a maximum possible pixel value.

Aspects of the invention provide an apparatus for HDR video processing.The apparatus includes a processing device (e.g., FPGA or ASIC) and aplurality of image sensors coupled to the processing device. Theapparatus is configured to stream pixel values from each of theplurality of image sensors in a frame independent-manner through apipeline on the processing device. The pipeline includes a kerneloperation that identifies saturated pixel values and a merge module tomerge the pixel values to produce an HDR image. The apparatus may use orbe connected to a single lens and one or more beamsplitters. Theplurality of image sensors includes at least a high exposure (HE) sensorand a middle exposure (ME) sensor. The HE sensor, the ME sensor, thelens and the beamsplitters may be arranged to receive an incoming beamof light and split the beam of light into at least a first path thatimpinges on the HE sensor and a second path that impinges on the MEsensor. Preferably, the beamsplitter directs a majority of the light tothe first path and a lesser amount of the light to the second path. Inpreferred embodiments, the first path and the second path impinge on theHE and the ME sensor, respectively, to generate images that areoptically identical but for light level.

The kernel operation may operate on pixel values as they stream fromeach of the plurality of image sensors by examining, for any given pixelon the HE sensor, values from a neighborhood of pixels surrounding thegiven pixel, finding saturated values in the neighborhood of pixels, andusing information from a corresponding neighborhood on the ME sensor toestimate a value for the given pixel. The pipeline may include—in theorder in which the pixel values flow: a sync module to synchronize thepixel values as the pixel values stream onto the processing device fromthe plurality of image sensors; the kernel operation; the merge module;a demosaicing module; and a tone-mapping operator as well as optionallyone or more of a color-conversion module; an HDR conversion module; andan HDR compression module. In certain embodiments, the apparatus alsoincludes a low exposure (LE) sensor. Preferably, the apparatus includesa sync module on the processing device, in the pipeline upstream of thekernel operation, to synchronize the pixel values as the pixel valuesstream onto the processing device from the plurality of image sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows steps of a method for producing real-time HDR video.

FIG. 2 shows an apparatus for HDR video processing.

FIG. 3 shows an arrangement for multiple sensors.

FIG. 4 shows a processing device on a real-time HDR video apparatus.

FIG. 5 shows operation of a sync module.

FIG. 6 illustrates how pixel values are presented to a kernel operation.

FIG. 7 shows an approach to modeling a pipeline.

FIG. 8 illustrates merging to avoid artifacts.

FIG. 9 shows a camera response curve used to adjust a pixel value.

FIG. 10 shows a color correction processes.

FIG. 11 illustrates a method for combined HDR broadcasting.

DETAILED DESCRIPTION

FIG. 1 shows steps of a method 101 for producing real-time HDR video.The method 101 includes receiving 107 light through the lens of animaging apparatus. One or more beamsplitters split 113 the light intodifferent paths that impinge upon multiple image sensors. Each imagesensor then captures 125 a signal in the form of a pixel value for eachpixel of the sensor. Where the sensor has, say, 1920×1080 pixels, thepixel values will stream off of the sensor to a connected processingdevice. The method includes streaming 129 pixel values 501 from each ofmultiple sensors in a frame independent-manner through a pipeline 231 ona processing device 219. The pipeline 231 includes a kernel operation135 that identifies saturated pixel values. The pixel values 501 aremerged 139. Typically, the merged image will be demosaiced 145 and thisproduces an HDR image that can be displayed, transmitted, stored, orbroadcast 151. In the described method 101, the multiple image sensorsall capture 125 images simultaneously through a single lens 311. Thepipeline 231 and kernel operation 135 may be provided by an integratedcircuit such as a field-programmable gate array or anapplication-specific integrated circuit. Each of the image sensors mayinclude a color filter array 307. In preferred embodiments, the method101 includes demosaicing 145 the HDR image after the merging step 139.The multiple image sensors preferably capture images that are opticallyidentical except for light level.

A feature of the invention is that the pixel values 501 are pipelineprocessed in a frame-independent manner. Sequences of pixel values 501are streamed 129 through the processing device 219 and merged 139without waiting to receive pixel values 501 from all pixels on the imagesensors. This means that the obtaining 125, streaming 129, and merging139 steps may be performed by streaming 129 the sequences of pixelvalues 501 through the pipeline 231 on the processing device 219 suchthat no location on the processing device 219 stores a complete image.Because the pixel values are streamed through the pipeline, the finalHDR video signal is produced in real-time. An apparatus 201 performingsteps of the method 101 thus provides the function of a real-time HDRvideo camera. Real-time means that HDR video from the camera may bedisplayed essentially simultaneously as the camera captures the scene(e.g., at the speed that the signal travels from sensor to display minusa latency no greater than a frame of film). There is no requirement forpost-processing the image data and no requirement to capture, store,compare, or process entire “frames” of images.

The output is an HDR video signal because the method 101 and theapparatus 201 use multiple sensors at different exposure levels tocapture multiple isomorphic images (i.e., identical but for light level)and merge them. Data from a high exposure (HE) sensor are used whereportions of an image are dim and data from a mid-exposure (ME) (orlower) sensor(s) are used where portions of an image are more brightlyilluminated. The method 101 and apparatus 201 merge the HE and ME (andoptionally LE) images to produce an HDR video signal. Specifically, themethod 101 and the apparatus 201 identify saturated pixels in the imagesand replace those saturated pixels with values derived from sensors of alower exposure. In preferred embodiments, a first pixel value from afirst pixel on one of the image sensors is identified as saturated if itis above some specified level, for example at least 90% of a maximumpossible pixel value.

FIG. 2 shows an apparatus 201 for HDR video processing. The apparatus201 includes a processing device 219 such as a field-programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC). Aplurality of image sensors 265 are coupled to the processing device 219.The apparatus 201 is configured to stream pixel values 501 from each ofthe plurality of image sensors 265 in a frame independent-manner througha pipeline 231 on the processing device 219. The pipeline 231 includes akernel operation 413 that identifies saturated pixel values 501 and amerge module to merge the pixel values 501 to produce an HDR image.

The kernel operation 413 operates on pixel values 501 as they streamfrom each of the plurality of image sensors 265 by examining, for agiven pixel on the HE sensor 213, values from a neighborhood 601 ofpixels surrounding the given pixel, finding saturated values in theneighborhood 601 of pixels, and using information from a correspondingneighborhood 601 on the ME sensor 211 to estimate a value for the givenpixel.

Various components of the apparatus 201 may be connected via a printedcircuit board 205. The apparatus 201 may also include memory 221 andoptionally a processor 227 (such as a general-purpose processor like anARM microcontroller). Apparatus 201 may further include or be connectedto one or more of an input-output device 239 or a display 267. Memorycan include RAM or ROM and preferably includes at least one tangible,non-transitory medium. A processor may be any suitable processor knownin the art, such as the processor sold under the trademark XEON E7 byIntel (Santa Clara, Calif.) or the processor sold under the trademarkOPTERON 6200 by AMD (Sunnyvale, Calif.). Input/output devices accordingto the invention may include a video display unit (e.g., a liquidcrystal display or LED display), keys, buttons, a signal generationdevice (e.g., a speaker, chime, or light), a touchscreen, anaccelerometer, a microphone, a cellular radio frequency antenna, portfor a memory card, and a network interface device, which can be, forexample, a network interface card (NIC), Wi-Fi card, or cellular modem.The apparatus 201 may include or be connected to a storage device 241.The plurality of sensors are preferably provided in an arrangement thatallows multiple sensors 265 to simultaneously receive images that areidentical except for light level.

FIG. 3 shows an arrangement for the multiple sensors 265. The multiplesensors preferably include at least a high exposure (HE) sensor 213 anda middle exposure (ME) sensor 211. Each image sensor may have its owncolor filter array 307. The color filter arrays 307 may operate as aBayer filter such that each pixel receives either red, green, or bluelight. As is known in the art, a Bayer filter includes a repeating gridof red, green, blue, green filters such that a sequence of pixel valuesstreaming from the sensor corresponds to values for red, green, blue,green, red, green, blue, green, red, green, blue, green, . . . etc.

As shown in FIG. 3, the apparatus 201 may also include or be opticallyconnected to a lens 311 and at least one beamsplitter 301. The HE sensor213, the ME sensor 211, the lens 311 and the at least one beamsplitter301 are arranged to receive an incoming beam of light 305 and split thebeam of light 305 into at least a first path that impinges and HE sensor213 and a second path that impinges on the ME sensor 211. In a preferredembodiment, the apparatus 201 uses a set of partially-reflectingsurfaces to split the light from a single photographic lens 311 so thatit is focused onto three imaging sensors simultaneously. In a preferredembodiment, the light is directed back through one of the beamsplittersa second time, and the three sub-images are not split into red, green,and blue but instead are optically identical except for their lightlevels. This design, shown in FIG. 3, allows the apparatus to captureHDR images using most of the light entering the camera.

In some embodiments, the optical splitting system uses two uncoated,2-micron thick plastic beamsplitters that rely on Fresnel reflections atair/plastic interfaces so their actual transmittance/reflectance (T/R)values are a function of angle. Glass is also a suitable materialoption. In one embodiment, the first beamsplitter 301 is at a 45. angleand has an approximate T/R ratio of 92/8, which means that 92% of thelight from the camera lens 311 is transmitted through the firstbeamsplitter 301 and focused directly onto the high-exposure (HE) sensor213. The beamsplitter 301 reflects 8% of the light from the lens 311upwards (as shown in FIG. 3), toward the second uncoated beamsplitter319, which has the same optical properties as the first but ispositioned at a 90. angle to the light path and has an approximate T/Rratio of 94/6.

Of the 8% of the total light that is reflected upwards, 94% (or 7.52% ofthe total light) is transmitted through the second beamsplitter 319 andfocused onto the medium-exposure (ME) sensor 211. The other 6% of thisupward-reflected light (or 0.48% of the total light) is reflected backdown by the second beamsplitter 319 toward the first beamsplitter 301(which is again at 45.), through which 92% (or 0.44% of the total light)is transmitted and focused onto the low-exposure (LE) sensor 261. Withthis arrangement, the HE, ME and LE sensors capture images with 92%,7.52%, and 0.44% of the total light gathered by the camera lens 311,respectively. Thus a total of 99.96% of the total light gathered by thecamera lens 311 has been captured by the image sensors. Therefore, theHE and ME exposures are separated by 12.2× (3.61 stops) and the ME andLE are separated by 17.0× (4.09 stops), which means that thisconfiguration is designed to extend the dynamic range of the sensor by7.7 stops.

This beamsplitter arrangement makes the apparatus 201 light efficient: anegligible 0.04% of the total light gathered by the lens 311 is wasted.It also allows all three sensors to “see” the same scene, so all threeimages are optically identical except for their light levels. Of course,in the apparatus of the depicted embodiment 201, the ME image hasundergone an odd number of reflections and so it is flipped left-rightcompared to the other images, but this is fixed easily in software. Inpreferred embodiments, the three sensors independently stream incomingpixel values directly into a pipeline that includes a synchronizationmodule. This synchronization module can correct small phasediscrepancies in data arrival times to the system from multiple sensors.

Thus it can be seen that the beamsplitter 301 directs a majority of thelight to the first path and a lesser amount of the light to the secondpath. Preferably, the first path and the second path impinge on the HEsensor 213 and the ME sensor 211, respectively, to generate images thatare optically identical but for light level. In the depicted embodiment,the apparatus 201 includes a low exposure (LE) sensor.

In preferred embodiments, pixel values stream from the HE sensor 213,the ME sensor 211, and the LE sensor 261 in sequences directly to theprocessing device 219. Those sequences may be not synchronized as theyarrive onto the processing device 219.

As shown by FIG. 3, the method 101 may include receiving 107 incominglight through the lens 311 and splitting 113 the light via at least onebeamsplitter 301 onto the multiple image sensors, wherein at least 95%of the incoming beam of light 305 is captured by the multiple imagesensors.

The apparatus 201 (1) captures optically-aligned, multiple-exposureimages simultaneously that do not need image manipulation to account formotion, (2) extends the dynamic range of available image sensors (byover 7 photographic stops in our current prototype), (3) is inexpensiveto implement, (4) utilizes a single, standard camera lens 311, and (5)efficiently uses the light from the lens 311.

The method 101 preferably (1) combines images separated by more than 3stops in exposure, (2) spatially blends pre-demosaiced pixel data toreduce unwanted artifacts, (3) produces HDR images that areradiometrically correct, and (4) uses the highest-fidelity (lowestquantized-noise) pixel data available. The apparatus 201 can work with avariety of different sensor types and uses an optical architecture basedon beamsplitters located between the camera lens and the sensors.

FIG. 4 shows the processing device 219 on the apparatus 201. As noted,the processing device 219 may be provided by one or more FPGA, ASIC, orother integrated circuit. Pixel values from the sensors stream throughthe pipeline 231 on the processing device 219. The pipeline 231 in theprocessing device 219 includes—in the order in which the pixel values501 flow: a sync module 405 to synchronize the pixel values 501 as thepixel values 501 stream onto the processing device 219 from theplurality of image sensors 265; the kernel operation 413; the mergemodule 421; a demosaicing module 425; and a tone-mapping operator 427.The pipeline 231 may include one or more auxiliary module 431 such as acolor-correction module; an HDR conversion module; and an HDRcompression module.

FIG. 5 shows operation of the sync module 405 to synchronize the pixelvalues 501 as the pixel values 501 stream onto the processing device 219from the plurality of image sensors 265. As depicted in FIG. 5, HE_1pixel value and ME_1 pixel value are arriving at the sync module 405approximately simultaneously. However, HE_2 pixel value will arrive latecompared to ME_2, and the entire sequence of LE pixel values will arrivelate. The sync module 405 can contain small line buffers that circulatethe early-arriving pixel values and release them simultaneous with thecorresponding later-arriving pixel values. The synchronized pixel valuesthen stream through the pipeline 231 to the kernel operation 413.

FIG. 6 illustrates how the pixel values are presented to the kerneloperation 413. The top part of FIG. 6 depicts the HE sensor 213. Eachsquare depicts one pixel of the sensor 213. A heavy black box with awhite center is drawn to illustrate a given pixel 615 for considerationand a neighborhood 601 of pixels surrounding the given pixel 615. Theheavy black box would not actually appear on a sensor 213 (such as aCMOS cinematic camera sensor)—it is merely drawn to illustrate what theneighborhood 601 includes and to aid understanding how the neighborhood601 appears when the sequences 621 of pixel values 501 are presented tothe kernel operation 413.

The bottom portion of FIG. 6 shows the sequences 621 of pixel values asthey stream into the kernel operation 413 after the sync module 405.Pixel values 501 from the neighborhood 601 of pixels on the sensor 213are still “blacked out” to aid illustration. The given pixel 615 underconsideration can be spotted easily because it is surrounded on eachside by two black pixels from the row of pixels on the sensor. There aretwo sequences 621, one of which comes from the depicted HE sensor 213and one of which originates at the ME sensor 211.

Streaming the pixel values 501 through the kernel operation 413 includesexamining values from a neighborhood 601 of pixels surrounding a firstpixel 615 on the HE sensor 213, finding saturated values in theneighborhood 601 of pixels, and using information from a correspondingneighborhood 613 from the ME sensor 211 to estimate a value for thefirst pixel 615. This will be described in greater detail below. Toaccomplish this, the processing device must make comparisons betweencorresponding pixel values from different sensors. It may be useful tostream the pixel values through the kernel operation in a fashion thatplaces the pixel under consideration 615 adjacent to each pixel from theneighborhood 601 as well as adjacent to each pixel from thecorresponding neighborhood on another sensor.

FIG. 7 shows an approach to modeling the circuit so that the pipelineplaces the current pixel 615 adjacent to each of the following pixelvalues: a pixel value from 1 to the right on the sensor 213, a pixelvalue from 2 pixels to the right on sensor 213, a pixel value from 1 tothe left, and pixel value from two to the left. As shown in FIG. 7, dataflows into this portion of the pipeline and is copied four additionaltimes. For each copy, a different and specific amount of delay is addedto the main branch. The five copies all continue to flow in parallel.Thus, a simultaneous snapshot across all five copies covers the givencurrent pixel value 615 and the other pixel values from the neighborhood601. In this way, pixel values on either side of the pixel currentlybeing processed can be used in that processing step, along with thepixel currently being processed. Thus the processing device cansimultaneously read and compare the pixel value of the given pixel tothe value of neighbors. The approach illustrated in FIG. 7 can beextended for comparisons to upper and lower neighbors, diagonalneighbors, and pixel values from a corresponding neighborhood on anothersensor. Thus in some embodiments, streaming 129 the pixel values 501through the kernel operation 413 includes streaming 129 the pixel values501 through a path 621 within the processing device 219 that momentarilyplaces a value from the first pixel proximal to each value originatingfrom the neighborhood 601 of pixels.

The neighborhood comparisons may be used in determining whether to use areplacement value for a saturated pixel and what replacement value touse. An approach to using the neighborhood comparisons is discussedfurther down after a discussion of the merging. A replacement value willbe used when the sequences 621 of pixel values 501 are merged 139 by themerge module 421. The merging 139 step excludes at least some of thesaturated pixel values 501 from the HDR image.

Previous algorithms for merging HDR images from a set of LDR images withdifferent exposures typically do so after demosaicing the LDR images andmerge data pixel-by-pixel without taking neighboring pixel informationinto account.

To capture the widest dynamic range possible with the smallest number ofcamera sensors, it is preferable to position the LDR images furtherapart in exposure than with traditional HDR acquisition methods. Priorart methods yield undesired artifacts because of quantization and noiseeffects, and those problems are exacerbated when certain tone mappingoperators (TMOs) are applied. Those TMOs amplify small gradientdifferences in the image to make them visible when the dynamic range iscompressed, amplifying merging artifacts as well.

FIG. 8 illustrates an approach to merging that reduces artifacts (e.g.,compared to the weighting factor used in a merging algorithm in Debevecand Malik, 1997, Recovering high dynamic range radiance maps fromphotographs, Proceedings of ACM SIGGRAPH 1997:369-378, incorporated byreference). The “HE sensor”, “ME sensor”, and “LE sensor” bars in FIG. 8present the range of scene illumination measured by the three sensors

For illustration, the system is simplified with 4-bit sensors (asopposed to the 12-bit sensors as may be used in apparatus 201), whichmeasure only 16 unique brightness values and the sensors are separatedby only 1 stop (a factor of 2) in exposure. Since CMOS sensors exhibitan approximately linear relationship between incident exposure and theiroutput value, the values from the three sensors are graphed as a linearfunction of incident irradiance instead of the traditional logarithmicscale.

Merging images by prior art algorithms that always use data from allthree sensors with simple weighting functions, such as that of Debevecand Malik, introduces artifacts. In the prior art, data from each sensoris weighted with a triangle function as shown by the dotted lines, sothere are non-zero contributions from the LE sensor at low brightnessvalues (like the sample illumination level indicated), even though thedata from the LE sensor is quantized more coarsely than that of the HEsensor.

Methods of the invention, in contrast, use data from the higher-exposuresensor as much as possible and blend in data from the next darker sensorwhen near saturation.

FIG. 8 shows that the LE sensor measures the scene irradiance morecoarsely than the other two sensors. For example, the HE sensor maymeasure 4 different pixel values in a gradient before the LE sensorrecords a single increment. In addition, there is always some smallamount of noise in the pixel values, and an error of ±1 in the LE sensorspans a 12 value range in the HE sensor for this example. AlthoughDebevec and Malik's algorithm blends these values together, the method101 and apparatus 201 use pixel values from only the longest-exposuresensor (which is less noisy) wherever possible, and blend in the nextdarker exposure when pixels approach saturation.

In certain embodiments, the method 101 and apparatus 201 not onlyexamine individual pixels when merging the LDR images, but also takeinto account neighboring pixels 601 (see FIG. 6) that might provideadditional information to help in the de-noising process.

One aspect of merging 139 according to the invention is to use pixeldata exclusively from the brightest, most well-exposed sensor possible.Therefore, pixels from the HE image are used as much as possible, andpixels in the ME image are only used if the HE pixel is close tosaturation. If the corresponding ME pixel is below the saturation level,it is multiplied by a factor that adjusts it in relation to the HE pixelbased on the camera's response curve, given that the ME pixel receives12.2× less irradiance than the HE pixel.

FIG. 9 shows a camera response curve 901 used to obtain a factor foradjusting a pixel value. In a three-sensor embodiment, when the HEsensor is above the saturation level, and if the corresponding ME pixelis above the saturation level, then a similar process is applied to thesame pixel in the low-exposure LE image.

It may be found that merging by a “winner take all” approach thatexclusively uses the values from the HE sensor until they becomesaturated and then simply switches to the next sensor results in bandingartifacts where transitions occur. To avoid such banding artifacts, themethod 101 and apparatus 201 transition from one sensor to the next byspatially blending pixel values between the two sensors. To do this, themethod 101 and apparatus 201 scan a neighborhood 601 around the pixel615 being evaluated (see FIG. 6). If any neighboring pixels in thisregion are saturated, then the pixel under consideration may be subjectto pixel crosstalk or leakage, and the method 101 and apparatus 201 willestimate a value for the pixel based on its neighbors in theneighborhood 601.

The method 101 and apparatus 201 perform merging 139 prior todemosaicing 145 the individual Bayer color filter array images becausedemosaicing can corrupt colors in saturated regions. For example, abright orange section of a scene might have red pixels that aresaturated while the green and blue pixels are not. If the image isdemosaiced before being merged into HDR, the demosaiced orange colorwill be computed from saturated red-pixel data and non-saturatedgreen/blue-pixel data. As a result, the hue of the orange section willbe incorrectly reproduced. To avoid these artifacts, the method 101 andapparatus 201 perform HDR-merging prior to demosaicing.

Since the images are merged prior to the demosaicing step, the method101 and apparatus 201 work with pixel values instead of irradiance. Toproduce a radiometrically-correct HDR image, the method 101 andapparatus 201 match the irradiance levels of the HE, ME, and LE sensorsusing the appropriate beamsplitter transmittance values for each pixelcolor, since these change slightly as a function of wavelength. Althoughthe method 101 and apparatus 201 use different values to match each ofthe color channels, for simplicity the process is explained with averagevalues. A pixel value is converted through the camera response curve901, where the resulting irradiance is adjusted by the exposure levelratio (average of 12.2× for HE/ME), and this new irradiance value isconverted back through the camera response curve 901 to a new pixelvalue.

FIG. 9 shows the 3-step HDR conversion process to match the irradiancelevels of the HE, ME, and LE sensors. The HDR conversion process may bedone for all HE pixel values (from 1 through 4096, for example), toarrive at a pixel-ratio curve, which gives the scaling factor forconverting each ME pixel's value to the corresponding pixel value on theHE sensor for the same irradiance. In practice, separate pixel-ratiocurves are calculated for each color (R,G,B) in the Bayer pattern. Whencomparing pixel values between HE and ME images (or between ME and LEimages), a simple multiplier may be used, or the pixel-ratio curves maybe used as lookup tables (LUTs), to convert HE pixel values less than4096 into ME pixel values, or vice versa. When the HE pixel values aresaturated, the pixel-ratio curve is extended using the last valueobtained there (approximately 8).

The camera response curve 901 can be measured by taking a set ofbracketed exposures and solving for a monotonically-increasing functionthat relates exposure to pixel value (to within a scale constant in thelinear domain).

FIG. 9 shows the curve computed from the raw camera data, although acurve computed from a linear best-fit could also be used.

FIG. 9 gives a camera response curve that shows how the camera convertsscene irradiance into pixel values. To compute what the ME pixel valueshould be for a given HE value, the HE pixel value (1) is firstconverted to a scene irradiance (2), which is next divided by our HE/MEattenuation ratio of 12.2. This new irradiance value (3) is convertedthrough the camera response curve into the expected ME pixel value (4).Although this graph is approximately linear, it is not perfectly sobecause it is computed from the raw data, without significant smoothingor applying a linear fit. With the irradiance levels of the three imagesmatched, the merging 139 may be performed.

In an illustrative example of merging 139, two registered LDR images(one high-exposure image IHE and a second medium-exposure image IME) areto be merged 139 into an HDR image IHDR . The merging 139 starts withthe information in the high-exposure image IHE and then combines in datafrom the next darker-exposure image IME, as needed. To reduce thetransition artifacts described earlier, the method 101 and apparatus 201work on each pixel location (x, y) by looking at the information fromthe surrounding (2k+1)×(2k+1) pixel neighborhood 601, denoted as N(x,y).

In some embodiments as illustrated in FIG. 6, the method 101 andapparatus 201 use a 5×5 pixel neighborhood 601 (k=2), and define a pixelto be saturated if its value is greater than some specific amount, forexample 90% of the maximum pixel value (4096 e.g., where sensor 213 is a12-bit CMOS sensor).

In certain embodiments, the merging 139 includes a specific operationfor each of the four cases for the pixel 615 on sensor 213 and itsneighborhood 601 (see FIG. 6):

Case 1: The pixel 615 is not saturated and the neighborhood 601 has nosaturated pixels, so the pixel value is used as-is.

Case 2: The pixel 615 is not saturated, but the neighborhood 601 has 1or more saturated pixels, so blend between the pixel value at IHE(x, y)and the one at the next darker-exposure IME(x, y) depending on theamount of saturation present in the neighborhood.

Case 3: The pixel 615 is saturated but the neighborhood 601 has 1 ormore non-saturated pixels, which can be used to better estimate a valuefor IHE(x,y): calculate the ratios of pixel values in the ME imagebetween the unsaturated pixels in the neighborhood and the center pixel,and use this map of ME ratios to estimate the actual value of thesaturated pixel under consideration.

Case 4: The pixel 615 is saturated and all pixels in the neighborhood601 are saturated, so there is no valid information from thehigh-exposure image, use the ME image and set IHDR(x, y)=IME(x, y).

When there are three LDR images, the process above is simply repeated ina second iteration, substituting IHDR for IHE and ILE for IME. In thismanner, data is merged 139 from the higher exposures while workingtoward the lowest exposure, and data is only used from lower exposureswhen the higher-exposure data is at or near saturation.

This produces an HDR image that can be demosaiced 145 and converted frompixel values to irradiance using a camera response curve similar to thatof FIG. 9 accounting for all 3 color channels. The final HDR full-colorimage may then be tone mapped (e.g., with commercial software packagessuch as FDRTools, HDR Expose, Photomatix, etc.)

The apparatus 201 may be implemented using three Silicon ImagingSI-1920HD high-end cinema CMOS sensors mounted in a camera body. Thosesensors have 1920×1080 pixels (5 microns square) with a standard Bayercolor filter array, and can measure a dynamic range of around 10 stops(excluding noise). The sensors are aligned by aiming the camera at smallpinhole light sources, locking down the HE sensor and then adjustingsetscrews to align the ME and LE sensors.

The camera body may include a Has selblad lens mount to allow the use ofhigh-performance, interchangeable commercial lenses. For beamsplitters,the apparatus may include uncoated pellicle beamsplitters, such as theones sold by Edmund Optics [part number NT39-482]. The apparatus 201 mayperform the steps of the method 101. Preferably, the multiple imagesensors include at least a high exposure (HE) sensor 213 and a middleexposure (ME) sensor 211, and the merging includes using HE pixel values501 that are not saturated and ME pixel values 501 corresponding to thesaturated pixel values. The multiple sensors may further include a lowexposure (LE) sensor 261, and the method 101 may include identifyingsaturated pixel values 501 originating from both the HE sensor 213 andthe ME sensor 211. Because the pixel values stream through a pipeline,it is possible that at least some of the saturated pixel values 501 areidentified before receiving values from all pixels of the multiple imagesensors at the processing device 219 and the method 101 may includebeginning to merge 139 portions of the sequences while still streaming129 later-arriving pixel values 501 through the kernel operation 413.

It is understood that optical components such as beamsplitters, lenses,or filters—even if labeled “spectrally neutral”—may have slightwavelength-dependent differences in the amounts of light transmitted.That is, each image sensor may be said to have its own “color correctionspace” whereby images from that sensor need to be corrected out of thatcolor correction space to true color. The optical system can becalibrated (e.g., by taking a picture of a calibration card) and a colorcorrection matrix can be stored for each image sensor. The HDR videopipeline can then perform the counter-intuitive step of adjusting thepixel values from one sensor towards the color correction of anothersensor—which may in some cases involve nudging the colors away from truecolor. This may be accomplished by multiplying a vector of RGB valuesfrom the one sensor by the inverse color correction matrix of the othersensor. After this color correction to the second sensor, the streamsare merged, and the resulting HDR video signal is color corrected totruth (e.g., by multiplying the RGB vectors by the applicable colorcorrection matrix). This color correction process accounts for spectraldifferences of each image sensor.

FIG. 10 shows a color correction processes 1001 by which the HDRpipeline can correct for differences in spectral characteristics of eachof the multiple sensors. To correct for the slight wavelength-dependentdifferences among the sensors, relationships between electron input andelectron output can be measured experimentally using known inputs. Bycomputing a correction factor for each sensor, the information detectedby the sensors can be corrected prior to further processing. Thus, insome embodiments, the pipeline 231 includes modules for colorcorrection. The steps of a color correction process may be applied atmultiple locations along the pipeline, so the color correction may beimplemented via specific modules at different locations on the FPGA.Taken together, those modules may be referred to as a color correctionmodule that implements the color correction process 1001.

The color correction process 1001 converts one sensor's data from itscolor correction space to the color correction space of another sensor,before merging the images from the two sensors. The merged image datacan then be converted to the color correction space of a third sensor,before being combined with the image data from that third sensor. Theprocess may be repeated for as many sensors as desired. After allsensors' images have been combined, the final combined image may bedemosaiced 145 and then may be color corrected to truth.

The color correction process 1001 allows images from multiple sensors tobe merged, in stages where two images are merged at a time, in a waythat preserves color information from one sensor to the next. Forexample purposes, in FIG. 10, the HE pixel values from the HE sensor aremerged with the ME pixel values from the ME sensor. The result ofmerging is then merged with the LE pixel values from the LE sensor.

The basic principle guiding the color correction process 1001 is tofirst convert a dark image to the color correction space of the nextbrightest image, and then to merge the two “non-demosaiced” (or ColorFilter Array [CFA] Bayer-patterned) images together.

The color correction process 1001, for an apparatus 201 with an MEsensor, an LE sensor, and an SE sensor, includes three general phases:an SE color correction space (CCS) phase, ME color correction spacephase, and LE color correction space phase. The color correction processfirst begins with the SE color correction space phase, which comprisesfirst demosaicing 1045 the LE pixel values and then transforming 1051the resulting vectors into the color correction space of the ME image.The demosaicing process 1045 yields a full-color RGB vector value foreach pixel.

After it has been demosaiced 1045, the LE image data is next transformed1045 into the ME color correction space. The purpose is to match thecolor of the LE pixels (now described by RGB vectors) to the color ofthe ME array (with all of the ME array's color imperfections). Toperform the transformation 1051, the LE RGB vectors are transformed 1051by a color correction matrix. For example, Equations 1-3 show how to usethe color correction matrices to correct the color values for the HE,ME, and LE sensors, respectively. Equation 1 shows how to use the colorcorrection matrix to correct the color values of the HE sensor, wherethe 3×3 matrix coefficients, including values A1-A9, representcoefficients selected to strengthen or weaken the pixel value, and anRGB matrix (RLE , GLE , and BLE) represents the demosaiced RGB outputsignal from the LE sensor. In some cases, the 3×3 matrix coefficientscan be derived by comparing the demosaiced output against expected (orso-called “truth”) values. For example, the 3×3 matrix coefficients canbe derived by least-squares polynomial modeling between the demosaicedRGB output values and reference values from a reference color chart(e.g., a Macbeth chart). Similarly, Equation 2 shows how to use thecolor correction matrix to correct the color values of the ME sensor,where the RGB matrix (RME , GME , and BME) represents the demosaiced RGBoutput signal from the ME sensor, and Equation 3 shows how to use thecolor correction matrix to correct the color values of the SE sensor,where the RGB matrix (RME, GME, and BME) represents the demosaiced RGBoutput values from the SE sensor.

$\begin{matrix}{{{correcting}\mspace{14mu} {SE}\mspace{14mu} {pixel}\mspace{14mu} {values}\mspace{14mu} {{using}\mspace{14mu}\lbrack A\rbrack}},{{{the}\mspace{14mu} {Color}\mspace{14mu} {Correction}\mspace{14mu} {Matrix}\mspace{14mu} {for}\mspace{14mu} {the}\mspace{14mu} {LE}\mspace{14mu} {{{sensor}\begin{bmatrix}A_{1} & A_{2} & A_{3} \\A_{4} & A_{5} & A_{6} \\A_{7} & A_{8} & A_{9}\end{bmatrix}}\begin{bmatrix}R_{LE} \\G_{LE} \\B_{LE}\end{bmatrix}}} = {{\lbrack A\rbrack \begin{bmatrix}R_{LE} \\G_{LE} \\B_{LE}\end{bmatrix}} = \begin{bmatrix}R_{truth} \\G_{truth} \\B_{truth}\end{bmatrix}}}} & {{{Equation}\mspace{14mu} 1}\;} \\{{{correcting}\mspace{14mu} {ME}\mspace{14mu} {pixel}\mspace{14mu} {values}\mspace{14mu} {{using}\mspace{14mu}\lbrack B\rbrack}},{{{the}\mspace{14mu} {Color}\mspace{14mu} {Correction}\mspace{14mu} {Matrix}\mspace{14mu} {for}\mspace{14mu} {the}\mspace{14mu} {ME}\mspace{14mu} {{{sensor}\begin{bmatrix}B_{1} & B_{2} & B_{3} \\B_{4} & B_{5} & B_{6} \\B_{7} & B_{8} & B_{9}\end{bmatrix}}\begin{bmatrix}R_{ME} \\G_{ME} \\B_{ME}\end{bmatrix}}} = {{\lbrack B\rbrack \begin{bmatrix}R_{ME} \\G_{ME} \\B_{ME}\end{bmatrix}} = \begin{bmatrix}R_{truth} \\G_{truth} \\B_{truth}\end{bmatrix}}}} & {{Equation}\mspace{14mu} 2} \\{{{correcting}\mspace{14mu} {SE}\mspace{14mu} {pixel}\mspace{14mu} {values}\mspace{14mu} {{using}\mspace{14mu}\lbrack C\rbrack}},{{{the}\mspace{14mu} {Color}\mspace{14mu} {Correction}\mspace{14mu} {Matrix}\mspace{14mu} {for}\mspace{14mu} {the}\mspace{14mu} {SE}\mspace{14mu} {{{sensor}\begin{bmatrix}C_{1} & C_{2} & C_{3} \\C_{4} & C_{5} & C_{6} \\C_{7} & C_{8} & C_{9}\end{bmatrix}}\begin{bmatrix}R_{SE} \\G_{SE} \\B_{SE}\end{bmatrix}}} = {{\lbrack C\rbrack \begin{bmatrix}R_{SE} \\G_{SE} \\B_{SE}\end{bmatrix}} = \begin{bmatrix}R_{truth} \\G_{truth} \\B_{truth}\end{bmatrix}}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

To convert an image from a first color correction space (CCS) to asecond color correction space, the color correction matrices from one ormore sensors can be used. This process may be referred to as convertingbetween color correction spaces or calibrating color correction spaces.Neither the first color correction space nor the second color correctionspace accurately reflects the true color of the captured image. Thefirst and the second color correction space both have inaccuracies, andthose inaccuracies are, in general, different from one another. Thus RGBvalues from each sensor must be multiplied by a unique color correctionmatrix for those RUB values to appear as true colors.

The present invention includes a method for converting an image from theLE sensor's color correction space to the ME sensor's color correctionspace and is illustrated in Equation 4 below:

$\begin{matrix}{{{converting}\mspace{14mu} {LE}\mspace{14mu} {pixel}\mspace{14mu} {values}\mspace{14mu} {from}\mspace{14mu} {LE}}{{{color}\mspace{14mu} {correction}\mspace{14mu} {space}\mspace{14mu} {to}\mspace{14mu} {ME}\mspace{14mu} {correction}\mspace{14mu} {{{{space}\begin{bmatrix}R_{SE} \\G_{SE} \\B_{SE}\end{bmatrix}}\lbrack C\rbrack}\lbrack B\rbrack}^{- 1}} = \begin{bmatrix}R_{ME} \\G_{ME} \\B_{ME}\end{bmatrix}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

In Equation 4, the LE sensor's pixel values (R, G, B) are multiplied bythe LE sensor's correction matrix. [C], and then multiplied by theinverse of the MME sensor's correction matrix, [B]. The result is a setof pixel values (R, G, B) that are in the ME sensor's color correctionspace.

Methods of the invention allow matching of the color correction space ofthe second sensor to the color correction space of the first sensor sothat the images from the two sensors may be accurately combined, ormerged. The method for applying all the inaccuracies of the second colorcorrection space to the first color correction space, prior to combiningimages from the two into an HDR image, is previously unknown. Typicalmethods for combining data from multiple CFA sensors rely oncolor-correcting each sensor's data to the “truth” values measured froma calibrated color card, prior to combining the images. This isproblematic in an HDR system, where it is known that the brightersensor's image will have significant portions that are saturated, whichsaturated portions should actually have been utilized from the darkersensor's image when combining. Color correcting an image that has colorinformation based on saturated pixels will cause colors to bemisidentified. Therefore, in an HDR system, color-correcting thebrighter image (for example, to “truth” color values), prior tocombining images, will lead to colors being misidentified because of theuse of saturated pixel data in creating colors from a mosaic-patternedimage. For this reason, we specify that (1) the darker image have itscolor information transformed to match the color space of the brighterimage, (2) this transformed darker image be combined with the brighterimage, and then (3) the final combined image be color-transformed to“truth” color values.

The solution provided in the present invention avoids thissaturated-pixel color misidentification problem by performing the stepsof [(a) demosaic 1045, (b) color correct 1051 & (c) mosaic 1057] datafrom the darker sensor, thereby ensuring all data is accurately returnedto its non-demosaiced state prior to the step of merging the darkersensor's data with the brighter sensor's data.

Furthermore, prior to merging the images from two sensors, the presentinvention matches the color correction spaces of the two sensors. Thistransformation ensures that the two images (from the first and secondcolor correction space sensors) can be accurately merged,pixel-for-pixel, in non-demosaiced format. It may at first seemcounterintuitive to change the color correction space of one sensor tomatch the color correction space of a second sensor, especially when thesecond sensor's color correction space is known to differ from the“true” color correction space. However, it is an important feature inensuring that (1) the brighter sensor's color information not bedemosaiced prior to merging, and (2) the color data from both sensors ismatched together, prior to merging the images. The color correctionprocess 1001 uses matrices that may themselves be implemented as kernelsin the pipeline 231 on the processing device 219. Thus the colorcorrection process 1001 is compatible with an HDR pipeline workflowbecause the kernels are applied as they receive the pixel values.

After the LE information is transformed 1051 from the LE colorcorrection space to the ME color correction space, the transformedvalues are mosaiced 1057 (i.e., the demosaicing process is reversed).The transformed scalar pixel values are now comparable with theBayer-patterned scalar ME pixel values detected by the ME sensor, andthe process 1001 includes merging 1061 of ME and HE non-demosaiced(i.e., scalar) sensor data.

The merged non-demosaiced image within the ME color correction space isthen demosaiced 1067. This demosaicing 1064 is similar to thedemosaicing 1045 described above, except the CFA pixel values undergoingthe demosaicing process are now associated with the ME color correctionspace. The demosaicing 1067 produces RGB vectors in the ME color space.Those RGB vectors are transformed 1071 into the HE color space whilealso being color corrected ([B][A]−1[RGB]). Equation 2 shows how to usethe color correction matrix to correct the color values of the MEsensor. The color corrected ME information is transformed 1071 from theME color correction space to the HE color correction space bymultiplying the ME color correction matrix by the inverse of the SEcolor correction matrix.

After the ME information is transformed 1071 from the ME colorcorrection space to the HE color correction space, the transformedvectors are mosaiced 1075 (i.e., the demosaicing process is reversed).This allows the transformed ME CFA Bayer-patterned pixel values to merge1079 with the HE pixel values detected by the HE sensor. At this pointin the color correction process 1001, the transformed color informationdetected by the HE and ME sensors is now calibrated to match the colorinformation detected by the HE sensor. This newly merged color valuedata set now represents color values within the HE color correctionspace 205.

FIG. 11 illustrates a method 1301 for combined broadcasting of highdynamic range (HDR) video with standard dynamic range (SDR) video. Themethod 1301 provides for streaming HDR and SDR video. The method 1301includes detecting 125—using an array of sensors 165—informationrepresenting a series of images, processing 1309 the information, andtransmitting 1321 the information for HDR and LDR display with less thanone frame of delay between detection and transmission.

After the color processing and tone-mapping, the pipeline has producedan HDR video signal. At this stage, the pipe can include a module forsubtraction 1315 that, in real-time, subtracts the SDR signal from theHDR signal (HDR−SDR=residual). What flows from the subtraction module isa pair of streams—the SDR video signal and the residual signal.Preferably, all of the color information is in the SDR signal. At thisstage the HDR signal may be subject to HDR compression by a suitableoperation (e.g., JPEG or similar). The pair of streams includes the8-bit SDR signal and the compressed HDR residual signal, which providesfor HDR display. This dual signal is broadcast over a communicationnetwork and may in-fact be broadcast over television networks, cellularnetworks, or the Internet. A device that receives the signal displaysthe video according to the capacity of that device. An SDR displaydevice will “see” the 8-bit SDR signal and display a video at a dynamicrange that is standard, which signal has also had a certain TMO appliedto it. An HDR display device will decompress the residuals and combinethe dual streams into an HDR signal and display HDR video.

Thus, the method 1301 and the apparatus 201 may be used for real-timeHDR video capture as well as for the simultaneous delivery of HDR andSDR output in a single transmission. The processing 1309 may include theworkflow from the processing device 219 to video (broadcast) output. Themethod 1301 and the apparatus 201 provide for real-time processing andcomplementary HDR/SDR display using features described herein such asmultiple sensors all obtaining an isomorphic image through a single lensand streaming the resulting pixel values through a pipeline to replacesaturated pixels in a merged HDR video signal. The method 101 and theapparatus 201 each captures video information using an array of sensors,processes that video information in real-time, and transmits the videoinformation in real-time in a HDR and LDR compatible format.

INCORPORATION BY REFERENCE

References and citations to other documents, such as patents, patentapplications, patent publications, journals, books, papers, webcontents, have been made throughout this disclosure. All such documentsare hereby incorporated herein by reference in their entirety for allpurposes.

EQUIVALENTS

Various modifications of the invention and many further embodimentsthereof, in addition to those shown and described herein, will becomeapparent to those skilled in the art from the full contents of thisdocument, including references to the scientific and patent literaturecited herein. The subject matter herein contains important information,exemplification and guidance that can be adapted to the practice of thisinvention in its various embodiments and equivalents thereof.

What is claimed is:
 1. A method for producing real-time HDR video, themethod comprising: streaming pixel values from each of multiple sensorsin a frame independent-manner through a pipeline on a processing device,wherein the pipeline includes a kernel operation that identifiessaturated pixel values; and merging the pixel values to produce an HDRimage.
 2. The method of claim 1, wherein the merging step excludes atleast some of the saturated pixel values from the HDR image.
 3. Themethod of claim 2, wherein the multiple image sensors all capture imagessimultaneously through a single lens.
 4. The method of claim 3, furthercomprising receiving incoming light through the lens and splitting thelight via at least one beamsplitter onto the multiple image sensors,wherein at least 95% of the light gathered by the imaging lens iscaptured by the multiple image sensors.
 5. The method of claim 4,wherein the multiple image sensors include at least a high exposure (HE)sensor and a middle exposure (ME) sensor, and wherein merging thesequences includes using HE pixel values that are not saturated and MEpixel values corresponding to the saturated pixel values.
 6. The methodof claim 5, wherein the multiple sensors further include a low exposure(LE) sensor, and the method includes identifying saturated pixel valuesoriginating from both the HE sensor and the ME sensor.
 7. The method ofclaim 5, wherein streaming the pixel values through the kernel operationincludes examining values from a neighborhood of pixels surrounding afirst pixel on the HE sensor, finding saturated values in theneighborhood of pixels, and using information from a correspondingneighborhood on the ME sensor to estimate a value for the first pixel.8. The method of claim 7, wherein at least some of the saturated pixelvalues are identified before receiving values from all pixels of themultiple image sensors at the processing device.
 9. The method of claim8, further comprising beginning to merge portions of the sequences whilestill streaming later-arriving pixel values through the kerneloperation.
 10. The method of claim 8, wherein sequences of pixel valuesare streamed through the processing device and merged without waiting toreceive pixel values from all pixels on the image sensors.
 11. Themethod of claim 10, wherein the processing device comprises one selectedfrom the group consisting of a field-programmable gate array and anapplication-specific integrated circuit, and further wherein each of theimage sensors comprises a color filter array, and the method furthercomprises demosaicing the HDR imaging after the merging.
 12. The methodof claim 11, wherein multiple image sensors capture images that areoptically identical except for light level.
 13. The method of claim 11,wherein streaming the pixel values through the kernel operation includesstreaming the pixel values through a path within the processing devicethat momentarily places a value from the first pixel proximal to eachvalue originating from the neighborhood of pixels.
 14. The method ofclaim 11, wherein the obtaining, streaming, and merging steps areperformed by streaming the sequences of pixel values through a pipelineon the processing device such that no location on the processing devicestores a complete image.
 15. The method of claim 5, wherein a firstpixel value from a first pixel on one of the image sensors is identifiedas saturated if it is at least 90% of a maximum possible pixel value.16. An apparatus for HDR video processing, the apparatus comprising: aprocessing device; a plurality of image sensors coupled to theprocessing device, wherein the apparatus is configured to stream pixelvalues from each of the plurality of image sensors in aframe-independent manner through a pipeline on the processing device,wherein the pipeline includes a kernel operation that identifiessaturated pixel values and a merge module to merge the pixel values toproduce an HDR image.
 17. The apparatus of claim 16, further comprisinga lens and at least one beamsplitter.
 18. The apparatus of claim 17,wherein the plurality of image sensors include at least a high exposure(HE) sensor and a middle exposure (ME) sensor.
 19. The apparatus ofclaim 18, wherein the HE sensor, the ME sensor, the lens and the atleast one beamsplitter are arranged to receive an incoming beam of lightand split the beam of light into at least a first path that impinges andHE sensor and a second path that impinges on the ME sensor.
 20. Theapparatus of claim 19, wherein the beamsplitter directs a majority ofthe light to the first path and a lesser amount of the light to thesecond path.